<p><!doctype html></p>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script type="text/javascript" async 
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-MML-AM_CHTML">
</script>
<!-- SYNTAX HIGHLIGHTING CLASSES  -->
<style type="text/css">
body {
    font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
    text-align:justify;
    margin: 0 8%;
    font-size: 14px;
        background-color:white; 
}

a {
   color: #1e6bb8;
       text-decoration: none;
}
a:hover {
       text-decoration: underline;
}

h1, h2, h3, h4, h5, h6 {
    line-height: 1.5;
}

h1, h1 > a, h1 > a:link {
        margin:0.67em 0;
        padding: 0;
        font-family: ;
        color:#17365D;
        font-size: 22px;
        }

h1 > a:hover, h1 > a:hover{
color:#345A8A;
} 

h2, h2 > a, h2 > a, h2 > a:link {
        margin:14px 0px 2px 0px;
        padding: 0;
        color:#345A8A;
        font-size: 18px;
        font-weight:bold;
        }

h3, h3 > a,h3 > a, h3 > a:link,h3 > a:link {
        margin:14px 0px 0px 0px;
        padding: 0;
        color:#4F81BD;
        font-size: 16px;
        font-weight:bold;
        }

h4 {
        margin:10px 0px 0px 0px;
        padding: 0;
        font-family: ;
        font-size: 16px;
        color:#4F81BD;
        font-weight:bold;
        font-style:italic;
        }

h5  {
        font-size: 14px;
        font-weight:normal;
        color:#4F81BD;
        }

h6  {
        font-size:14px;
        font-weight:normal;
        font-style:italic;
        color:#4F81BD;
        }

p {
        font-weight:normal;
        font-size:14px;
        line-height: 16px;
        text-align:justify;
        text-justify:inter-word;
        margin:0 0 14px 0;
        }

.author {display:block;text-align:center;font-size:16px;margin-bottom:3px;}
.date {display:block;text-align:center;font-size:12px;margin-bottom:3px;}
.center, #center {
    display: block;
    margin-left: auto;
    margin-right: auto;
    -webkit-box-shadow: 0px 0px 2px rgba( 0, 0, 0, 0.5 );
    -moz-box-shadow: 0px 0px 2px rgba( 0, 0, 0, 0.5 );
    box-shadow: 0px 0px 2px rgba( 0, 0, 0, 0.5 );

    padding: 0px;
    border-width: 0px;
    border-style: solid;
    cursor:-webkit-zoom-in;
    cursor:-moz-zoom-in;
    }

td > p {padding:0; margin:0;}


header {
        font-size:28px;
        padding-bottom:5px; 
        margin:0;
        padding-top:150px; 
        font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        text-align:center;
        display:block;
        font-weight:bold;
        }



table {
        border-collapse: collapse;
        border-bottom:1px solid black;
        padding:5px;
        margin-top:5px;
        margin-bottom: 16px;

}
.tble {
        display:block;
        margin-top: 10px;
        margin-bottom: 0px;
        margin-bottom: 0px;
}

.tblecenter {
        display:block;
        margin-top: 10px;
        margin-bottom: 0px;
        margin-bottom: 0px;
        text-align:center;
}

span.tblecenter + table, span.tble + table, span.tble + img {
        margin-top: 2px;
}

.figure, .caption {
        text-align: center
}

th {
        border-bottom:1px solid black;
        border-top:1px solid black;
        padding-right:20px;
}

td {
        padding-right:20px;
}

pre code {
       color: #6D6D6D;
       font-size: 10px;
       padding-top: 0;
       padding-bottom: 0;
       margin-top: 0;
       margin-bottom: 0;
      font-family: "menlo-regular","Consolas", "Liberation Mono", Menlo, Courier, monospace;
   display: block;
}

.sh_stata {
   font-size: 0.7rem;
       padding-top: 0;
       padding-bottom: 0;
       margin-top: 0;
       margin-bottom: 0;
   display: block;
   line-height: 1.5;
}

pre > code {
   padding: 0;
       margin: 0;
       word-break: normal;
   display: block;
}

.sh_stata + pre code {
    margin:-10px 0 0 -10px;
    line-height:8px;
}

blockquote {
       padding: 0 1rem;
       margin-left: 0;
       color: #819198;
       border-left: 0.3rem solid #dce6f0;
}

img {
        margin: 5px 0 5px 0;
        padding: 0px;
        cursor:-webkit-zoom-in;
        cursor:-moz-zoom-in;
        display:inline-block;
        clear: both;
        }

hr {
       height: 2px;
       margin: 1rem 0;
       background-color: #eff0f1;
       border: 0; 
}

ul li {
       font-size:14px;
}

</style>
<script type="text/javascript" src='http://haghish.com/statax/Statax.js'></script>
</head>
<body>
<span class="date">21 Dec 2018</span>
<h1><a href="#web-scraping-nba-data-into-stata" id="web-scraping-nba-data-into-stata">Web scraping NBA data into Stata</a></h1>
<ul>
<li>Author: Kevin Crow, Senior Software Developer</li>
<li><a href="https://blog.stata.com/2018/10/10/web-scraping-nba-data-into-stata/">https://blog.stata.com/2018/10/10/web-scraping-nba-data-into-stata/</a></li>
</ul>
<p>This is a replication of a post from Stata blog to test MarkDoc package. Unfortunately, the example was not replicable on Stata 15.1 and Windows 10. Though, this post is based on a user-written program.</p>
<hr />
<p>Since our intern, Chris Hassell, finished <strong>nfl2stata</strong> earlier than expected, he went ahead and created another command to web scrape <a href="https://stats.nba.com">https://stats.nba.com</a> for data on the NBA. The command is <strong>nba2stata</strong>. To install the command type</p>
<pre class="sh_stata">  . net install http://www.stata.com/users/kcrow/nba2stata, replace</pre>
<pre><code>      checking nba2stata consistency and verifying not already installed...
      all files already exist and are up to date.
</code></pre>
<p>When Chris first wrote the command, I knew I wanted to look at how the three-point shot has changed the way the game is played. For example, I can find the best three-point shooter from last season.</p>
<pre class="sh_stata">  . nba2stata playerstats _all, season(2017) seasontype(reg) stat(season) clear</pre>
<pre><code>      Processing x/548 requests

      .........x.........x.........x.........x.........50
      .........x.........x.........x.........x.........100
      .........x.........x.........x.........x.........150
      .........x.........x.........x.........x.........200
      .........x.........x.........x.........x.........250
      .........x.........x.........x.........x.........300
      .........x.........x.........x.........x.........350
      .........x.........x.........x.........x.........400
      .........x.........x.........x.........x.........450
      .........x.........x.........x.........x.........500
      .........x.........x.........x.........x........
      664 observation(s) loaded
</code></pre>
<hr />
<pre class="sh_stata">  . gsort -threepointfieldgoalsmade</pre>
<pre class="sh_stata">  . list playername teamname threepointfieldgoalsmade in 1/10</pre>
<pre><code>           +----------------------------------------------------+
           |      playername                teamname   three~de |
           |----------------------------------------------------|
        1. |    James Harden         Houston Rockets        265 |
        2. |     Paul George   Oklahoma City Thunder        244 |
        3. |      Kyle Lowry         Toronto Raptors        238 |
        4. |    Kemba Walker       Charlotte Hornets        231 |
        5. |   Klay Thompson   Golden State Warriors        229 |
           |----------------------------------------------------|
        6. | Wayne Ellington              Miami Heat        227 |
        7. |  Damian Lillard   Portland Trailblazers        227 |
        8. |     Eric Gordon         Houston Rockets        218 |
        9. |   Stephen Curry   Golden State Warriors        212 |
       10. |      Joe Ingles               Utah Jazz        204 |
           +----------------------------------------------------+
</code></pre>
<hr />
<p>Or I can check a player’s regular-season three-point percentage for the last five years.</p>
<pre class="sh_stata">  . nba2stata playerstat "Dirk", stat(season) seasontype(reg) clear</pre>
<pre><code>      28 observation(s) loaded
</code></pre>
<pre class="sh_stata">  . gsort -playerage </pre>
<pre class="sh_stata">  . list playername playerage threepointfieldgoalpercentage in 1/5</pre>
<pre><code>           +-------------------------------------+
           |    playername   playe~ge   three~ge |
           |-------------------------------------|
        1. | Dirk Nowitzki         40       .409 |
        2. | Dirk Nowitzki         40        .25 |
        3. | Dirk Nowitzki         39       .378 |
        4. | Dirk Nowitzki         38       .368 |
        5. | Dirk Nowitzki         37        .38 |
           +-------------------------------------+
</code></pre>
<hr />
<p>Or I can see how three-point percentage affects your favorite team’s chance of winning.</p>
<pre class="sh_stata">  . nba2stata teamstats "HOU", season(2017) stat(game) seasontype(reg) clear</pre>
<pre><code>      82 observation(s) loaded
</code></pre>
<pre class="sh_stata">  . keep if threepointfieldgoalpercentage > .35</pre>
<pre><code>      (35 observations deleted)
</code></pre>
<pre class="sh_stata">  . tab winloss</pre>
<pre><code>       Win / loss |      Freq.     Percent        Cum.
      ------------+-----------------------------------
                L |          4        8.51        8.51
                W |         43       91.49      100.00
      ------------+-----------------------------------
            Total |         47      100.00
</code></pre>
<p><strong>nba2stata</strong> is great if you are planning on doing pro basketball analysis. Although this command looks identical to <strong>nfl2stata</strong>, it is not. The command works quite differently.</p>
<hr />
<h2><a href="#web-scraping-json" id="web-scraping-json">Web scraping JSON</a></h2>
<p>In our last blog post, we talked about web scraping the <a href="https://www.nfl.com">https://www.nfl.com</a> and extracting the data from the HTML pages. The NBA data are different. You can access the data via JSON objects from <a href="https://stats.nba.com">https://stats.nba.com</a>. JSON is a lightweight data format. This data format is easy to parse; therefore, we don’t have a scrape command for these data. We scrape and load these data on the fly.</p>
<p>The NBA’s copyright is similar to that of the NFL; you can use a personal copy of the data on your own personal computer. If you “use, display or publish” anything using these data, you must include “a prominent attribution to <a href="http://www.nba.com“">http://www.nba.com“</a>. Another difference is that the NBA data stored on <a href="http://stats.nba.com">http://stats.nba.com</a> can go as far back as the 1960s, depending on the team.</p>
<hr />
<h3><a href="#command" id="command">Command</a></h3>
<p>There are only four subcommands to nba2stata, though we could have developed more. Chris had to go back to school.</p>
<ul>
<li>
<p>To scrape player statistics data into Stata, use</p>
<pre><code> nba2stata playerstats name_pattern [, playerstats_options]
</code></pre>
</li>
<li>
<p>To scrape player profile data into Stata, use</p>
<pre><code> nba2stata playerprofile name_pattern [, playerprofile_options]
</code></pre>
</li>
<li>
<p>To scrape team statistics data into Stata, use</p>
<pre><code> nba2stata teamstats team_adv [, teamstats_options]
</code></pre>
</li>
<li>
<p>To scrape team roster data into Stata, use</p>
<pre><code> nba2stata teamroster team_adv [, teamroster_options]
</code></pre>
</li>
</ul>
<p>Just like with nfl2stata, you will need to use Stata commands like collapse, gsort, and merge to generate the statistics, sort the data, and merge two or more NBA datasets together to examine the data.</p>
<hr />
<h4><a href="#examples" id="examples">Examples</a></h4>
<p>One thing I’m always curious about is which college teams produce the most NBA players. This is easy to find out using <strong>nba2stata</strong>, <strong>collapse</strong>, and <strong>gsort</strong>.</p>
<pre class="sh_stata">  . nba2stata playerprofile "_all", clear</pre>
<pre><code>      Processing x/4351 requests

      .........x.........x.........x.........x.........50
      .........x.........x.........x.........x.........100
      .........x.........x.........x.........x.........150
      .........x.........x.........x.........x.........200
      .........x.........x.........x.........x.........250
      .........x.........x.........x.........x.........300
      .........x.........x.........x.........x.........350
      .........x.........x.........x.........x.........400
      .........x.........x.........x.........x.........450
      .........x.........x.........x.........x.........500
      .........x.........x.........x.........x.........550
      .........x.........x.........x.........x.........600
      .........x.........x.........x.........x.........650
      .........x.........x.........x.........x.........700
      .........x.........x.........x.........x.........750
      .........x.........x..java.lang.reflect.InvocationTargetException
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at com.stata.Javacall.load(Javacall.java:130)
              at com.stata.Javacall.load(Javacall.java:90)
      Caused by: java.lang.NumberFormatException: For input string: &quot;N&quot;
              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
              at java.lang.Integer.parseInt(Integer.java:580)
              at java.lang.Integer.parseInt(Integer.java:615)
              at com.google.gson.JsonPrimitive.getAsInt(JsonPrimitive.java:264)
              at com.stata.chassell.deserializers.PlayerProfileRowSetDeserializer.deserialize(PlayerProfileRowSetDeserializer.java:43)
              at com.stata.chassell.deserializers.PlayerProfileRowSetDeserializer.deserialize(PlayerProfileRowSetDeserializer.java:1)
              at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
              at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
              at com.google.gson.internal.bind.ArrayTypeAdapter.read(ArrayTypeAdapter.java:72)
              at com.google.gson.Gson.fromJson(Gson.java:927)
              at com.google.gson.Gson.fromJson(Gson.java:994)
              at com.google.gson.internal.bind.TreeTypeAdapter$GsonContextImpl.deserialize(TreeTypeAdapter.java:162)
              at com.stata.chassell.deserializers.PlayerProfileResultSetDeserializer.deserialize(PlayerProfileResultSetDeserializer.java:24)
              at com.stata.chassell.deserializers.PlayerProfileResultSetDeserializer.deserialize(PlayerProfileResultSetDeserializer.java:1)
              at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
              at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41)
              at com.google.gson.internal.bind.ArrayTypeAdapter.read(ArrayTypeAdapter.java:72)
              at com.google.gson.Gson.fromJson(Gson.java:927)
              at com.google.gson.Gson.fromJson(Gson.java:994)
              at com.google.gson.internal.bind.TreeTypeAdapter$GsonContextImpl.deserialize(TreeTypeAdapter.java:162)
              at com.stata.chassell.deserializers.PlayerProfileResponseDeserializer.deserialize(PlayerProfileResponseDeserializer.java:20)
              at com.stata.chassell.deserializers.PlayerProfileResponseDeserializer.deserialize(PlayerProfileResponseDeserializer.java:1)
              at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
              at com.google.gson.Gson.fromJson(Gson.java:927)
              at com.google.gson.Gson.fromJson(Gson.java:892)
              at com.google.gson.Gson.fromJson(Gson.java:841)
              at com.google.gson.Gson.fromJson(Gson.java:813)
              at com.stata.chassell.NBA2Stata.getPlayerProfile(NBA2Stata.java:1781)
              at com.stata.chassell.NBA2Stata.lambda$8(NBA2Stata.java:2729)
              at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
              at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
              at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
              at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
              at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
              at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
              at com.stata.chassell.NBA2Stata.doCommand(NBA2Stata.java:2726)
              ... 6 more
      r(5100);

      end of do-file
      r(5100);

      end of do-file

      r(5100);
</code></pre>
</body>
</html>
